1. Field of the invention
The invention relates to electronic devices, and, more particularly, to multiprocessor and digital signal processor frameworks and methods.
2. Background
The growth of the Internet coupled with high-speed network access has thrust real-time multimedia processing into the computing mainstream. Indeed, the proposed MPEG-21 standardization of multimedia framework architecture considers seven architectural elements: digital item definition, multimedia content representation, digital item identification and description, content management and usage, intellectual property management and protection, terminals and networks, and event reporting. Applications of such a architecture could be in creation of a photo album distributed over the computers of widely dispersed family members with connections over the Internet and with editing facilities or could be consumption of Internet streaming video with related simultaneous additional content available which needs resolution of quality of service (QoS) and terminal and network resources available issues prior to consumption.
MPEG-21 proposal standard interfaces aim to shield users from network and terminal installation, management, and implementation issues. Such architecture could include a global QoS manager in each terminal which controls terminal resource and prediction managers and interacts through APIs with a user plus with network resource and prediction managers; see FIG. 25 and FIG. 24 for general network architecture.
The need for a QoS manager within a terminal/network node (client/server) stems specifically from real-time service requirements of all streaming-media based applications. Streaming media applications have to deal with heterogeneous codecs (encoders/decoders) and filters with unique rendering deadlines. These applications should also be able to exploit and translate human perceptual characteristics to graceful degradations in the quality of service. They should be able to handle reasonable amounts of jitter in their processing and rendering cycles. For instance, in video applications, the frame rate for rendering has to be maintained at 30 frames/sec (fps), which translates to a frame period of 33 ms. The application, however, should be capable of withstanding limited instantaneous variations as negotiated with the server. Also, at 30 fps, human visual perception can withstand frame drops of about 6 frames/sec. The client application should again be capable of supporting a graceful degradation in performance (instantaneous dropping of frames) and maintain a steady-state of rendering within specific tolerances negotiated with the server. A QoS manager is the mechanism that provides the necessary functions and capabilities to realize such a real-time system.
Microsoft DirectShow provides an application framework on a general purpose processor for playback of multimedia streams from local files or Internet servers and for capture of multimedia streams from devices. DirectShow revolves about a filter graph made of pluggable filter components with a filter graph manager controlling the connection of filters and the stream's data flow through the filters. Applications control the filter graph by communication with the filter graph manager; see FIG. 26. Playback from an Internet simply requires the source filter be capable of reading from an Internet URL. A parser filter after the source filter can perform the parsing into audio, video, text, etc. streams; see FIG. 27. The filter graph manager controls and also searches for needed filters (e.g., renderer) in a registry with merit values for selection. The filters have pins for filter input/output. FGM supports media stream starting, pausing, duration of play, . . . by application/ActiveX controls calls to the filter graph manager which accesses methods of the filters, and the filter graph manager posts events from the filters to the application.
A filter is a COM object for performing a task; and a pin is a COM object created by a filter for unidirectional data stream to/from the filter; see FIG. 28. A filter has IBaseFilter interface which enumerates the pins, properties, et cetera and inherits from IMediaFilter which allow control of state processing such as running, pausing, and stopping, plus synchronization; called by FGM. Pin interface supports transfer of time-stamped data using shared memory or other resources, negotiate data formats at pin-to-pin connections, buffer management/allocation to minimize data copying and maximize data throughput. IQualityControl interface on output pins.
Data flow in filter graphs (protocols), including quality control data, originates in the renderer and flows upstream (or to QCM) through the filters until it finds a filter capable of media data flow change. For example, media sample protocol: how media samples are allocated and passed between filters. Quality management protocol how filter graph adapts dynamically to hardware and network conditions to degrade/improve performance gracefully.
Media sample data flow by either push or pull: source filter push by call IMemInputPin::Receive from the downstream filter, or downstream pull by IAsyncReader interface (IAsyncReader Transport). Media samples are data objects which support IMediaSample interface. Data from one filter to another is called “transport”, and there is support for local memory transport in DirectShow classes: input pin supports IMemInputPin interface and output pin. The IAsyncReader interface is for pull.
Notenboom U.S. Pat. No. 5,748,468 and Equator Technologies PCT published application WO 99/12097 each describes methods of allocating processor resources to multiple tasks. Notenboom considers a host processor plus coprocessor with tasks allocated to coprocessor resources according to a priority system. Equator Technologies schedules processor resources according to task time consumption with each task presenting at least one service level (processor resource consumption rate) supported, and the resource manager admits a task if sufficient resources for a supported service level exist.
Systems with two or more processors, each processor with its own operating system or BIOS, include systems with widely separated processors connected via the Internet and also systems with two or more processors integrated on the same semiconductor die, such as a RISC CPU plus one or more DSPs.
The XDAIS standard prescribes interfaces for algorithms which run on DSPs; this provides reusable objects. XDAIS requires an algorithm implement the standard interface IALG plus an extension for running the algorithm. XDAIS also requires compliance with certain flexibility rules such as relocatable code and naming conventions. A client application can manage an instance of the algorithm by calling into a table of function pointers. With the XDAIS standard/guidelines the algorithm developer is able to develop or convert an algorithm so that it is easier to plug into a DSP application.
FIG. 20 shows a diagram of how data flows through the processing elements of current heterogeneous systems. The data transactions are numbered 1 through 6 to show time ordering. For each transaction data must pass through the system bus under control of the Central Control Processor (CCP). The CCP initiates transactions by sending messages or triggers via the control paths to the various processing elements in the system.
Processing elements in FIG. 20 are shown as separate processors (e.g. DSPs, ASICs, GPPs, etc.) capable of running a defined set of tasks. That is why each is shown with its own memory. Processing elements can also be individual tasks running on the same processor.
In some cases, the same data must pass through the system bus multiple times (e.g. transactions 1 and 2, 3 and 4, and 5 and 6). In such systems data must pass through the system bus a total of 2+(2×n) times, or in this case 6 times. Each pass through the system bus and intervention by the CCP introduces data flow overhead and reduces overall system throughput.
Data flow overhead negatively impacts how much data can move through the system in a given time frame and thereby restricts the amount of data the system is capable of processing. Such a system would likely be performing fewer useful tasks than the sum of capabilities of its elements might otherwise indicate.